Web Survey Bibliography
In the literature on questionnaire design and survey methodology, pre-testing is mentioned as a way to evaluate questionnaires (i.e. investigate whether they work as intended) and control for measurement errors (i.e. assess data quality). As the American Statistical Association puts it (ASA, 1999, p. 11): “The questionnaire designer must understand the need to pretest, pretest, and then pretest some more.” Clark and Schober (1992, p. 29) indicate why this need to pre-test: “Surveyors cannot possibly write perfect questions, self-evident to each respondent, that never need clarification. And because they cannot, the answers will often be surprising.”
In this Ph.D. thesis I have tried to systematically describe my experiences with pre-test research at the Questionnaire Laboratory at Statistics Netherlands, a cognitive laboratory which started its work in 1992. This text is not aimed at a theoretical discussion of cognitive laboratory methods, but focuses on the application of these methods: setting-up and carrying-out pre-test research, analysing the data and presentation of the results.
The thesis starts with an introduction of cognitive laboratory research, including the history of the CASM (Cognitive Aspects of Survey Methodology) movement and the history of the Questionnaire Laboratory at Statistics Netherlands. The next two chapters address aspects of computer-assisted interviewing. Since at Statistics Netherlands most social-survey questionnaires are computerised, this sets the conditions for pre-test research at the Questionnaire Laboratory. Chapter 2 discusses computer-assisted interviewing in general; chapter 3 addresses the effects of computer-assisted interviewing on data quality. These three chapters are introductory to the chapters that follow, the actual core of the thesis, in which the application of cognitive laboratory methods, including several case studies, are discussed.
The methods used at the Questionnaire Laboratory at Statistics Netherlands are discussed in chapters 4 and 5. Chapter 4 presents an overview of pre-test methods. Expert appraisal, focus groups, in-depth interviewing (including follow-up probing, meaning-oriented probing, paraphrasing, targeted test questions, and vignettes), and behavioural coding are discussed from a practical point of view, i.e. how they are applied in the Questionnaire Laboratory. Computer-Assisted Qualitative Interviewing (CAQI) is discussed in chapter 5. The CAQI method has been developed at the Questionnaire Laboratory to pre-test computerised questionnaires. With CAQI a pre-test protocol is integrated in a computerised questionnaire to be tested.
The next four chapters present case studies of cognitive research in which the methods addressed in the chapter 4 and 5 have been applied. These chapters discuss the design and the results (i.e. identified problems in the questionnaire and recommendations for improvement) of these studies.
Chapter 10 concludes this thesis with a summary. A number of identified problems in the investigated questionnaires are: vague or unclear wording, complex syntax, long question, double-barrelled question, conflict with previous question(s), question asks for specific information that is not available by heart, difficult to come to an answer because of complex calculation, overlapping or missing response items. In the conclusion, the identified problems are related to design errors in the questionnaire.
igitur Homepage (abstract) / (full text)
Web survey bibliography (210)
- In search of best practices; 2017; Kappelhof, J. W. S.; Steijn, S.
- The perils of non-probability sampling; 2017; Bethlehem, J.
- Estimating the Impact of Measurement Differences Introduced by Efforts to Reach a Balanced Response...; 2017; Kappelhof, J. W. S.; De Leeuw, E. D.
- Data chunking for mobile web: effects on data quality; 2017; Lugtig, P. J.; Toepoel, V.
- Are Final Comments in Web Survey Panels Associated with Next-Wave Attrition?; 2016; McLauchlan, C.; Schonlau, M.
- Participation rates of childhood cancer survivors to self-administered questionnaires: a systematic...; 2016; Kilsdonk, E.; Wendel, E.; van Dulmen-den Broeder, E.; van Leeuwen, F.E.; Van Den Berg, M. H.; Jaspers...
- A look into the challenges of mixed-mode surveys; 2016; Klausch, L. T.
- Unintentional Mobile Respondents in Official Statis tics and Their Effect on Data Quality ; 2016; Bakker, J.
- Tracking the Representativeness of an Online Panel Over Time ; 2016; Klausch, L. T.; Scherpenzeel, A.
- Detecting careless respondents in web-based questionnaires: Which method to use?; 2016; Niesen, A. S. M.; Meijer, R. R.; Tendeiro, J. N.
- Establishing the accuracy of online panels for survey research; 2016; Bruggen, E.; van den Brakel, J.; Krosnick, J. A.
- Surveying End-of-Life Medical Decisions in France: Evaluation of an Innovative Mixed-Mode Data Collection...; 2016; Legleye, S; Pennec, S.; Monnier, A.; Stephan, A.; Brouard, N.; Bilsen, J.; Cohen, J.
- Adaptive survey designs to minimize survey mode effects – a case study on the Dutch Labor Force...; 2016; Calinescu, M.; Schouten, B.
- Reducing Underreports of Behaviors in Retrospective Surveys: The Effects of Three Different Strategies...; 2016; Lugtig, P. J.; Glasner, T.; Boeve, A.
- Sunday shopping – The case of three surveys; 2016; Bethlehem, J.
- Predictive inference for non-probability samples: a simulation study ; 2016; Buelens, B.; Burger, J.; van den Brakel, J.
- Does the Inclusion of Non-Internet Households in a Web Panel Reduce Coverage Bias?; 2016; Eckman, S.
- Internet Panels, Professional Respondents, and Data Quality; 2015; Matthijsse, S.; De Leeuw, E. D.; Hox, J.
- Effect of Web-Based Versus Paper-Based Questionnaires and Follow-Up Strategies on Participation Rates...; 2015; Kilsdonk, E.; van den Heuvel-Eibrink, M. M.; van Dulmen-den Broeder, E.; van der Pal, H. J. H.; van...
- Designing web surveys for the multi-device internet; 2015; de Bruijne, M.
- Self-identification of occupation in web surveys: requirements for search trees and look-up tables ; 2015; Tijdens, K. G.
- Tailored fieldwork design to increase representative household survey response: an experiment in the...; 2015; Luiten, A.; Schouten, B.
- Calendar Instruments in Retrospective Web Surveys; 2015; Glasner, T.; van der Vaart, W.; Dijkstra, W.
- Validating self-reported mobile phone use in adults using a newly developed smartphone application; 2015; Goedhart, G., Kromhout, H., Wiart, J., Vermeulen, R.
- Face-to-Face or Sequential Mixed-Mode Surveys Among Non-Western Minorities in the Netherlands: The Effect...; 2015; Kappelhof, J.
- Finding Item Nonresponse Patterns: Three Internet Survey Experiments Into the Effects of Nonresponse...; 2015; Van De Maat, J.
- The effectiveness of incentives on recruitment and retention rates: an experiment in a web survey; 2015; Mulder, J.; Douhou, S.
- Using WhatsApp as a Survey Tool; 2015; Ongena, Y. P.; Haan, M.
- The Effects of Adding a Mobile-Compatible Design to the American Life Panel; 2015; Toepoel, V.; Lugtig, P. J.; Amin, A.
- Adapting Grid Questions for Mobile Devices; 2015; de Bruijne, M.; Das, M.; van Soest, A.; Wijnant, A.
- Correlates of early and late responses to surveys in an online panel; 2015; Douhou, S.; Vis, C.
- Higher Item Nonresponse Rates Caused by Slider Scales in Web Surveys; 2015; Toepoel, V.; Funke, F.
- Coding Surveys on their Item Characteristics: Reliability Diagnostics; 2015; Bais, F.; Schouten, B.; Toepoel, V.
- Selection error in single- and mixed mode surveys of the Dutch general population; 2015; Hox, J., Klausch, L. T., Schouten, B.
- Investigating Response Quality in Mobile and Desktop Surveys: A Comparison of Radio Buttons, Visual...; 2014; Toepoel, V.; Funke, F.
- 640 Current trends in management of high-risk prostate cancer in Europe: Results of a web-based survey...; 2014; Briganti, A., Isbarn, H., Ost, P., Ploussard, G., Sooriakumaran, P., Van Den Bergh, R.C.N., Van Oort...
- Query on Data Collection for Social Surveys; 2014; Blanke, K., Luiten, A.
- Improving Response Rates and Questionnaire Design for Mobile Web Surveys; 2014; de Bruijne, M., Wijnant, A.
- Quality of physical therapy from a patient's perspective; factor analysis on web-based survey data...; 2014; Scholte, M., Calsbeek, H., Nijhuis-van der Sanden, M. W. G., Braspenning, J.
- Mining “Big Data” using Big Data Services ; 2014; Reips, U.-D., Matzat, U.
- Barriers and facilitators for participation in a preventive pelvic floor muscle training program from...; 2014; Albers-Heitner, P., Moossdorff-Steinhauser, H., Weemhoff, M., Nieman, F., Berghmans, B.
- Informing panel members about study results; 2014; Scherpenzeel, A., Toepoel, V.
- Targeting the bias – the impact of mass media attention on sample composition and representativeness...; 2014; Steinmetz, S., Oez, F., Tijdens, K. G.
- Exploring selection biases for developing countries - is the web a promising tool for data collection...; 2014; Tijdens, K. G., Steinmetz, S.
- Evaluating mixed-mode redesign strategies against benchmark surveys: the case of the Crime Victimization...; 2014; Klausch, L. T., Hox, J., Schouten, B.
- The quality of ego-centered social network data in web surveys: experiments with a visual elicitation...; 2014; Marcin, B., Matzat, U., Snijders, C.
- Measuring the very long, fuzzy tail in the occupational distribution in web-surveys; 2014; Tijdens, K. G.
- Social desirability is the same in offline, online, and paper surveys: A meta-analysis; 2014; Dodou, D., de Winter J. C. F.
- The impact of contact effort on mode-specific selection and measurement bias; 2014; Schouten, B., van der Laan, J., Cobben, F.
- Clicking vs. Dragging: Different Uses of the Mouse and Their Implications for Online Surveys; 2014; Sikkel, D., Steenbergen, R., Gras, S.